ReproHack @ Leiden University

A hands-on Reproducibility Hackathon



📅 30th November, 2019
⏰ 10 am - 5pm
📍 Leiden University Library

Welcome 👋

What’s a ReproHack?

We are all excited by the progress made by many authors to make their papers reproducible by publishing associated code and data.

We know how challenging it can be so we want to showcase the value of the practice, both for original authors and as a learning experience for those who attempt to reproduce the work.

During a ReproHack, participants attempt to reproduce published research of their choice from a list of proposed papers with publicly available associated code and data. Participants get to work with other people’s material in a low pressure environment and record their experiences on a number of key aspects, including reproducibility, transparency and reusability of materials. At the end of the day we regroup, share our experiences and give feedback to the authors.


It’s imperative to note that ReproHacks are by no means an attempt to criticise or discredit work. We see reproduction as beneficial scientific activity in itself, with useful outcomes for authors and valuable learning experiences for the participants and the research community as a whole.

We strive to make this event open and inclusive to all. As such we ask you to read our Code of Conduct. By participating, you are expected to uphold this code.



Join the event and reproduce ♻️

Register and save your spot by clicking here!

Join us at the ReproHack and get working with other people’s material.

Benefits to participants:
  • Practical experience in reproducibility with real published materials and the opportunity to explore different tools and strategies.

  • Inspiration from working with other people’s code and data.

  • An appreciation that reproducibility is non trivial but that opening up your work for more people to engage with is the best way to help improve it. An appreciation that reproducibility has community value beyond just the validation of the results. For example, access to such materials increases the potential for reuse and understanding of the work.

Benefits to the whole research community:
  • Assessment of how reproducible papers are ‘out of the box’.

  • Evaluation of how successful current practices are and for what purpose.

  • Identification of what works and where the most pressing weaknesses in our approaches are.







Propose a paper 📜

Click here to nominate your paper!

You’ve put a lot of effort into making your work reproducible. Now let people learn from and engage with it!

We invite nominations for papers that have both associated code and data publicly available. We also encourage analyses based on open source tools as we cannot guarantee participants will have access to specialised licenced software.


Benefits to authors:
  • Feedback on the reproducibility of your work.

  • Appreciation for your efforts in making your work reproducible.

  • Opportunity to engage others with your research.


Proposed papers:


1. Improving on Adjusted R-Squared

Karch, J. (2019, September 16). Improving on Adjusted R-Squared. https://doi.org/10.31234/osf.io/v8dz5

submitted by Julian D. Karch

Why should we attempt to reproduce this paper?

First, I think I came quite far in making it reproducible. The paper uses a simulation study. All the steps from the raw results of the simulation study to the final manuscript can be reproduced by clicking one button. This works because all code + all dependencies are stored online within a virtual machine that anybody can access. So, I think it is a quite good example for others to learn from. Second, the code running the simulation study itself is not included in this process because it would take far too long to run on one machine. I ran the code on a supercomputer. I would be interested in how people would try to reproduce such long-running code and whether they have feedback on how to improve sharing such long-running code.

Paper URL: https://doi.org/10.31234/osf.io/v8dz5

Data URL: https://doi.org/10.24433/CO.8023088.v1

Code URL: https://doi.org/10.24433/CO.8023088.v1

Useful programming skills: R

2. Spatial modelling of rice yield losses in Tanzania due to bacterial leaf blight and leaf blast in a changing climate

Spatial modelling of rice yield losses in Tanzania due to bacterial leaf blight and leaf blast in a changing climate. C. Duku, A. H. Sparks, S. J. Zwart. Climatic Change 135.3-4 (2016) pp. 569–583. Springer Nature. doi: 10.1007/s10584-015-1580-2

submitted by Adam Sparks

Why should we attempt to reproduce this paper?

This was my third attempt at making a paper fully reproducible. To date I it’s the most reproducible that I have published. I’m interested to know what stumbling blocks exist that I’m not aware of (aside from needing software like ArcGIS to fully rerun the complete analysis).

Paper URL: https://link.springer.com/article/10.1007/s10584-015-1580-2?wt_mc=internal.event.1.SEM.ArticleAuthorOnlineFirst

Data URL: https://figshare.com/articles/MICORDEA/1408501

Code URL: https://github.com/adamhsparks/MICCORDEA

Useful programming skills: R, Python, ArcGIS

3. Climate change may have limited effect on global risk of potato late blight.

Sparks, A. H., Forbes, G. A, Hijmans, R. J., & Garrett K. A. (2014). Climate change may have limited effect on global risk of potato late blight. Global Change Biology, doi:10.1111/gcb.12587.

submitted by Adam Sparks

Why should we attempt to reproduce this paper?

This is a two-for one. The repository contains code for companion papers, the model development and the model implementation and analysis. As the repository notes, some data are not freely available so I’ve made an effort to allow the paper to be replicated as best possible with what’s available.

Paper URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/gcb.12587

Data URL: https://figshare.com/articles/Supporting_files_for_Climate_change_may_have_limited_effect_on_global_risk_of_potato_late_blight/1066070

Code URL: https://github.com/adamhsparks/Global-Late-Blight-MetaModelling

Useful programming skills: R

4. Sea level regulated tetrapod diversity dynamics through the Jurassic/Cretaceous interval

Tennant, J. P., Mannion, P. D., & Upchurch, P. (2016). Sea level regulated tetrapod diversity dynamics through the Jurassic/Cretaceous interval. Nature Communications, 7, 12737.

submitted by Jon Tennant

Why should we attempt to reproduce this paper?

Because it’s a fun paper, involving dinosaurs! But one which I myself have also attempted to reproduce in the past, and struggled with. There are a few additional tweaks that might throw some people off too.

Paper URL: https://www.nature.com/articles/ncomms12737

Data URL: https://www.nature.com/articles/ncomms12737#supplementary-information

Code URL: https://www.nature.com/articles/ncomms12737#supplementary-information

Useful programming skills: R, Perl

5. Evaluation of App-Embedded Disease Scales for Aiding Visual Severity Estimation of Cercospora Leaf Spot of Table Beet

Del Ponte EM, Nelson SC, Pethybridge SJ (2019) Evaluation of App-Embedded Disease Scales for Aiding Visual Severity Estimation of Cercospora Leaf Spot of Table Beet. Plant disease 103:1347-1356. 10.1094/PDIS-10-18-1718-RE

submitted by Emerson M. Del Ponte

Why should we attempt to reproduce this paper?

There are data and code written in RMarkdown which allows to reproduce the entire analysis and plots shown of the paper. It also allows to generate HTML document, which is a nice interface that facilitates the reader to understand better why some procedures were adopted and how to run them.

Paper URL: https://apsjournals.apsnet.org/doi/10.1094/PDIS-10-18-1718-RE

Data URL: https://osf.io/ezxps/

Code URL: https://github.com/emdelponte/paper-estimate-app

Useful programming skills: R

6. Bayesian determination of the effect of a deep eutectic solvent on the structure of lipid monolayers

Phys. Chem. Chem. Phys., 2019,21, 6133-6141

submitted by Andrew McCluskey

Why should we attempt to reproduce this paper?

I believe this represents the only example of a reproducible paper from scattering data collected at Diamond Light Source (UK) and the Institute Laue-Langevin (France)

Paper URL: https://doi.org/10.1039/c9cp00203k

Data URL: https://doi.org/10.15125/BATH-00548

Code URL: https://doi.org/10.5281/zenodo.2577796

Useful programming skills: Python, make

7. A multiscale Bayesian inference approach to analyzing subdiffusion in particle trajectories

K. Hinsen and G.R. Kneller, J. Chem. Phys. 145, 151101 (2016)

submitted by Konrad Hinsen

Why should we attempt to reproduce this paper?

This is one of the very few papers in biomolecular simulation for which code and data are available and which should be reproducible. But it is also three years old, so it is an interesting test case for the longevity of reproducible research. The infrastructure software is available at http://www.activepapers.org/python-edition/ (with instructions for installation and use)

Paper URL: https://doi.org/10.1063/1.4965881

Data URL: https://doi.org/10.5281/zenodo.162171

Code URL: https://doi.org/10.5281/zenodo.162171

Useful programming skills: Python

8. Resolving the Measurement Uncertainty Paradox in Ecological Management

Memarzadeh, M., & Boettiger, C. (2019). Resolving the Measurement Uncertainty Paradox in Ecological Management. The American Naturalist, 193(5). https://doi.org/10.1086/702704

submitted by Carl Boettiger

Why should we attempt to reproduce this paper?

This will probably be a non-trivial example to reproduce, owing to: (1) long-running code, (2) dependency on external data sources, (3) possibly challenging software dependencies – both trivial ones (e.g. setting up custom fonts and plot themes) and critical ones (requires an external R package wrapping a C++ algorithm, not available on CRAN and can sometimes have interesting compiler issues, like when Apple decided to break the clang compiler in 10.0). Ideally one could just run the R code given in the appendix on your local R session, but that may take a bit of effort. We’ve tried to take steps to address those issues by providing caches of slow-running parts, providing a docker container, and providing sufficient annotations, but who knows!

Paper URL: https://doi.org/10.1086/702704

Data URL: NA

Code URL: https://github.com/boettiger-lab/pomdp-intro

Useful programming skills: R

9. Comparisons of Citizen Science Data-Gathering Approaches to Evaluate Urban Butterfly Diversity

Prudic KL, Oliver JC, Brown BV, Long EC. 2018. Comparisons of Citizen Science Data-Gathering Approaches to Evaluate Urban Butterfly Diversity. Insects. 9(4):E186. doi: 10.3390/insects9040186

submitted by Kathleen Prudic

Why should we attempt to reproduce this paper?

This is a fairly digestible paper with statistical analyses and data visualization that rely heavily on open data from citizen science projects.

Paper URL: https://doi.org/10.3390/insects9040186

Data URL: https://doi.org/10.5281/zenodo.1436741

Code URL: https://doi.org/10.5281/zenodo.1436741

Useful programming skills: R

10. Bivariate spatial point patterns in the retina: a reproducible review.

Eglen SJ (2016) Bivariate spatial point patterns in the retina: a reproducible review. Journal de la Société Française de Statistique 157:33–48.

submitted by Stephen Eglen

Why should we attempt to reproduce this paper?

Tell me what I should improve!

Paper URL: https://github.com/sje30/eglen2015

Data URL: NA

Code URL: NA

Useful programming skills: R

11. A data repository and analysis framework for spontaneous neural activity recordings in developing retina

https://doi.org/10.1186/2047-217X-3-3

submitted by Stephen Eglen

Why should we attempt to reproduce this paper?

Tell me what I can improve on; maybe think of other visualisations for data?

Paper URL: https://doi.org/10.1186/2047-217X-3-3

Data URL: NA

Code URL: http://www.damtp.cam.ac.uk/user/eglen/waverepo/

Useful programming skills: R

12. Population structure and phenotypic variation of Sclerotinia sclerotiorum from dry bean (Phaseolus vulgaris) in the United States

Kamvar ZN, Amaradasa BS, Jhala R, McCoy S, Steadman JR, Everhart SE. 2017. Population structure and phenotypic variation of Sclerotinia sclerotiorum from dry bean (Phaseolus vulgaris) in the United States. PeerJ 5:e4152 https://doi.org/10.7717/peerj.4152

submitted by Zhian N. Kamvar

Why should we attempt to reproduce this paper?

This paper is reproduced weekly in a docker container on continuous integration, but it is also set up to work via local installs as well. It would be interesting to see if it’s reproducible with a human operator who knows nothing of the project or toolchain.

Paper URL: https://peerj.com/articles/4152/

Data URL: https://osf.io/k8wtm

Code URL: https://github.com/everhartlab/sclerotinia-366

Useful programming skills: R, Make and knowledge of Docker containers

13. PREPRINT: Using digital epidemiology methods to monitor influenza-like illness in the Netherlands in real-time: the 2017-2018 season

Schneider P, Van Gool C, Spreeuwenberg P, Hooiveld M, Donker GA, Barnett DJ, Paget J. Using digital epidemiology methods to monitor influenza-like illness in the Netherlands in real-time: the 2017-2018 season. BioRxiv. 2018 Jan 1:440867.

submitted by Paul Schneider

Why should we attempt to reproduce this paper?

This preprint is an attempt to reproduce Google Flu Trend in the Netherlands.

The whole paper + code is meant to be easily reproducible and transferable to other countries and/or areas. If you are familiar with time series data, lasso regression and cross validation, the analysis should be straight forward.

If anyone is interested, I could also provide influenza data for other European countries.

Paper URL: https://www.biorxiv.org/content/10.1101/440867v1.full

Data URL: https://zenodo.org/record/1459862#.XQbNIG8vNPM

Code URL: https://zenodo.org/record/1459862#.XQbNIG8vNPM

Useful programming skills: R

14. Growth Dynamics of Independent Gametophytes of Pleurosoriopsis makinoi ( Polypodiaceae)

Bulletin of the National Museum of Nature and Science, Series B (Botany) 45: 77–86

submitted by Joel Nitta

Why should we attempt to reproduce this paper?

It uses the drake R package that should make reproducibility of R projects much easier (just run make.R and you’re done). However, it does depend on very specific package versions, which are provided by the accompanying docker image.

Paper URL: https://www.joelnitta.com/publication/2019-03-27_pleurosoriopsis/

Data URL: https://github.com/joelnitta/pleurosoriopsis

Code URL: https://github.com/joelnitta/pleurosoriopsis

Useful programming skills: R

15. Open Trade Statistics

@misc{open_trade_statistics_2019, title = {OTS BETA DASHBOARD}, url = {https://shiny.tradestatistics.io/}, author = {{Open Trade Statistics}}, publisher = {Open Trade Statistics}, year = {2019}, month = {Apr}, note = {Accessed: June 22, 2019} }

submitted by Mauricio Vargas

Why should we attempt to reproduce this paper?

The focus of the project is reproducibility. Here we show the differences to access data compared to similar initiatives: https://ropensci.org/blog/2019/05/09/tradestatistics/. Also, similar projects have obscure parts, while our exposes the code from raw data downloading to dashboard creation.

Paper URL: https://shiny.tradestatistics.io

Data URL: https://api.tradestatistics.io

Code URL: https://github.com/tradestatistics

Useful programming skills: R, Shiny

16. model4you: An R Package for Personalised Treatment Effect Estimation

Seibold, H., Zeileis, A. and Hothorn, T., 2019. model4you: An R Package for Personalised Treatment Effect Estimation. Journal of Open Research Software, 7(1), p.17. DOI: http://doi.org/10.5334/jors.219

submitted by Heidi Seibold

Why should we attempt to reproduce this paper?

I guess it could be a cool learning experience. The paper is written with knitr, uses a seed, is part of the R package it describes, was openly written using version control (SVN, R-Forge) and is available in an open access journal (@up_jors).

Paper URL: http://doi.org/10.5334/jors.219

Data URL: NA

Code URL: https://r-forge.r-project.org/scm/viewvc.php/pkg/model4you/inst/JORS/?root=partykit

Useful programming skills: R, knitr (LaTeX), version control (SVN)

17. Comparing theory-driven and data-driven attractiveness models using images of real women’s faces

Holzleitner et al. (In press). Comparing theory-driven and data-driven attractiveness models using images of real women’s faces, JEP:HPP.

submitted by Ben Jones

Why should we attempt to reproduce this paper?

Complex analyses over multiple variables. In press, so we can still fix errors ahead of publication!!

Paper URL: https://psyarxiv.com/vhc5k

Data URL: https://osf.io/jurcq/

Code URL: https://osf.io/jurcq/

Useful programming skills: R





Program 📅

10:00 Coffee and Tea ☕

10:30 Welcome

10:40 Presentation about tools for Reproducible Research by Dr. Anna Krystalli

11:30 Forming groups and start hackathon 👨🏻‍💻 👨🏼‍💻

12:30 Lunch buffet 🥗 🥙

14:30 2nd presentation (tbc)

15:00 Continue hacking 👨🏽‍💻👩🏼‍💻

16:30 Drinks and bites 🍻